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[A. Brief summary of the invention 

In this invention, we propose an efficient implementation of the film grain simulation algorithm 
described in [1,2,3]. In particular, we propose to limit the film grain generation process to an 
initialization step in which a pool of film grain blocks is formed. The pool of blocks could 
provide different film grain samples for different color components and for different intensity 
levels if required by the model used to simulate the film grain. 

This approach avoids the generation of film grain for blending with every processed block of 
pixels from the input image. As a result, it strongly reduces the computational complexity of 
the film grain simulation process. 

B. Keywords: list keywords or combinations of keywords to guide patent and literature searches. 
Underline the most important keywords. 

Film grain simulation, SEI message, JVT, MPEG-4 AVC, H.264. 

(C. Brief discussion of the pr oblem solved by the invention- ~ 

In previous disclosures [3,4], we proposed a method for encoding film grain as supplemental 
enhancement information for a coded sequence. More specifically, we proposed (1) to filter 
grain out of images before compression, then (2) transmit compressed video together with a 
message containing information about original grain, and finally (3) let the decoder restore 
original grainy appearance of images by simulating film grain based on the content of the 
message. This method provides large bit-rate savings for high quality applications where the 
film grain is to be preserved. However, it increases decoders complexity since film grain has 
to be reproduced and blended to the decoded images as specified by the transmitted 
information. 

According to this invention, we propose a method for simulating film grain in the frequency 
domain by mosaicing a series of pre-computed samples. This method avoids the generation 
of film grain for blending with every processed block of pixels from the input image. As a 
result, it strongly reduces the computational complexity of the film grain simulation process. 



D. Discussion of how you or others have implemented similar things in the past, including the 
manner in which others have attempted to solve the problem. Point out disadvantages and 
weaknesses in previous practice. Include literature references where available 

Film grain simulation has been implemented in several commercially available products as 
Cineon, trademark of Eastman Kodak Co., and Grain Surgery from Visual Infinity Inc.. These 
film grain simulation tools are commonly used in post-production to blend computer- 
generated objects into natural scenes with low constraints on complexity. 

The efficient implementation of film grain simulation algorithms became a topic of interest 
only recently since the adoption of proposal [2] in the Professional Amendment of the H.264 
standard. The standardization of an SEI message conveying film grain parameters will force 
all H.264 video decoders willing to support that SEI message to implement film grain 
simulation algorithms. However, at the best of our knowledge, no public literature exits 
describing efficient implementations of the specified algorithms. 



E. Description of the invention, including one or more practical embodiments of the invention in 
sufficient detail to allow one with ordinary skill in the art to practice the invention. Include 
schematic(s), flow chart(s) and or figures to clarify operation of the invention. Point out 
important features and items you believe to be new. State advantages of the invention and 
sacrifices, if any, made to achieve these advantages. Describe any experiments conducted and the 
results of those experiments 

BACKGROUND ART 

According to the technique described in [1,2,3], film grain can be simulated on a block basis 
as illustrated in Figure 1 . In a first step, the input image without film grain is decomposed in 
non-overlapping blocks of NxM pixels. Blocks are then read in raster-scan order until the 
whole image is scanned. Each of these blocks originates a block of film grain. Note that 
because the film grain characteristics may vary depending on the intensity level, the 
algorithm may select a different set of parameters depending on the average intensity 
measured on the extracted block. In a particular embodiment, the set of parameters will be 
conveyed in an SEI message as proposed in [1 ,2]. The set of film grain blocks generated 
through this process is stored and organized to form a film grain image that maps the size of 
the original input image. This composition process is what we have called mosaicing. 

Figure 2 depicts the steps of the process that results in the generation of a block of NxM film 
grain samples. The first step consists in generating a block of NxM random values. A 
Discrete Cosine Transform (DCT) is then computed to obtain the corresponding NxM 
frequency coefficients. Note that other transforms could also be used within the same 
scheme. The frequency filtering of this set of coefficients, computed as specified in [6], 
controls the size of the simulated grain. Finally, the last step consists in computing the 
Inverse Discrete Cosine Transform (IDCT) of the modified set of coefficients. This process 
requires the specification of the noise deviation as well as the high and low cut frequencies 
that control the filtering process in the frequency domain. According to [1 ,2,3], such 
information could be conveyed in an SEI message for video coding applications. 

Film grain blending with the original image is then performed as illustrated in Figure 3. 
Because individual blocks of film grain have been generated independently, artifacts can be 
perceived at the transitions between blocks. Thus, an optional filtering stage is introduced to 
reduce the visual artifacts resulting from the film grain mosaicing. 



DESCRIPTION OF THE INVENTION 

In this invention, we propose an efficient implementation of the film grain simulation algorithm 
previously described. In particular, we propose to limit the film grain generation to a pool 
creation process in which a limited number of film grain blocks are computed. This approach 
avoids the generation of film grain for blending with every processed block of pixels from the 
input image. As a result, we strongly reduce the computational complexity of the film grain 
simulation process. 

Different embodiments of this invention are possible depending on whether the pool is 
created prior to the film grain insertion or whether it is created and further updated depending 
on the input image statistics. We describe some of these embodiments below. 



Pool creation pr c ss implement d as an initialization step 

The pool creation process, as illustrated in Figure 4, consists in generating a set of K blocks 
of film grain samples for each set of parameters in the SEI message. Let us remember that 
according to [3], different sets of parameters could be sent for different color components and 
different intensity levels. A particular implementation with K=1 will result in the lowest 
computational requirements, generating only one block of film grain per set of parameters. 
However, to avoid the creation of patterns, the use of a larger number of blocks or even the 
use of a different number of blocks for different intensity intervals is strongly recommended. 

Following the pool creation process, information from the SEI message is not further required 
to assist the mosaicing process that creates the film grain image. As illustrated in Figure 5, 
film grain generation is now avoided by directly selecting a pre-computed block of film grain 
from the pool. When more than one block is available for the same intensity level and color 
component, a selecting criterion should be specified. In a particular implementation, the 
system could pseudo-randomly select one of the blocks from the pool to avoid the creation of 
patterns when a reduced number of blocks is available. Along these lines, the mosaicing 
process could make use of transformed copies of the set of available film grain blocks in a 
similar approach to that described in [5]. 

According to this implementation, the pool creation process is completely decoupled from the 
film grain blending stage. Despite of the computational advantages of anticipating the film 
grain generation, this approach has the inconvenient of being unaware of whether the input 
images will make use of all the blocks stored in the pool. The following approach allows a 
better management of the available memory based on the statistics of the input images. 



Pool creation and update based on input image statistics 

In a particular embodiment, the pool creation process could be performed based on the input 
image statistics. In this case, the computation of film grain blocks is not done a priori on all 
the possible intensity intervals described by the SEI message. Instead, the pool creation 
and/or update are done depending on the input image statistics, as illustrated in Figure 6. 

Following this flowchart, when blocks in the pool are available, the algorithm avoids the 
computation of new film grain samples. However, if no blocks are available in the pool for a 
specific intensity interval, the algorithm proceeds to the computation of a new film grain block 
that becomes part of the pool. The algorithm may estimate that blocks are not available not 
only when the pool is empty, but also if the existing blocks have been recently used for 
blending. Other criteria for managing the pool creation and update process could also be 
envisaged. 

An algorithm in accordance with this invention could progressively create and/or update the 
pool depending on the statistics of the input images. For example, the algorithm could 
organize the pool in order to dispose of a larger number of blocks in to those intensity 
intervals that are most used, to avoid the creation of visual patterns. According to this 
strategy, the distribution of film grain blocks in the pool could be progressively adapted to 
match variations in the image statistics (the sequence darkens or lightens). 
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Figure 1. - Film grain simulation at the decoder end based on the mosaicing of blocks of NxM 
film grain samples generated by filtering in the frequency domain. 
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Figure 2. - Generation of NxM film grain samples in the frequency domain. 
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Figure 3. - Film grain blending at the decoder with an optional filtering stage introduced to 
reduce the visual artifacts resulting from the film grain mosaicing. 
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Figure 5. - Film grain simulation at the decoder end based on the mosaicing of pre-computed 
blocks of film grain samples as described by this invention. 
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Figure 6. - Pool creation and update based on the statistics of the input images. 
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Possible claims 

1. A method for simulating film grain by mosaicing a series of pre-computed samples. 

2. Claim 1 where the pre-computed film grain samples are organized in a pool. 

3. Claim 2 where the pool generation process is conceived as an initialization step in which film 
grain blocks are computed before blending. 

4. Claim 2 where the pool is dynamically created and/or updated depending on the input image 
statistics. 

5. Claim 2 where the pool provides different film grain samples for different color components 
and/or different intensity levels. 

6. Claim 2 where transformed versions of the computed blocks of film grain are also used in the 
mosaicing process. 



